RESUMEN
BackgroundMany serological assays to detect SARS-CoV-2 antibodies were developed during the COVID-19 pandemic. Differences in the detection mechanism of SARS-CoV-2 serological assays limited the comparability of seroprevalence estimates for populations being tested. MethodsWe conducted a systematic review and meta-analysis of serological assays used in SARS-CoV-2 population seroprevalence surveys, searching for published articles, preprints, institutional sources, and grey literature between January 1, 2020, and November 19, 2021. We described features of all identified assays and mapped performance metrics by the manufacturers, third-party head-to-head, and independent group evaluations. We compared the reported assay performance by evaluation source with a mixed-effect beta regression model. A simulation was run to quantify how biased assay performance affects population seroprevalence estimates with test adjustment. ResultsAmong 1807 included serosurveys, 192 distinctive commercial assays and 380 self-developed assays were identified. According to manufacturers, 28.6% of all commercial assays met WHO criteria for emergency use (sensitivity [Sn.] >= 90.0%, specificity [Sp.] >= 97.0%). However, manufacturers overstated the absolute values of Sn. of commercial assays by 1.0% [0.1, 1.4%] and 3.3% [2.7, 3.4%], and Sp. by 0.9% [0.9, 0.9%] and 0.2% [-0.1, 0.4%] compared to third-party and independent evaluations, respectively. Reported performance data was not sufficient to support a similar analysis for self-developed assays. Simulations indicate that inaccurate Sn. and Sp. can bias seroprevalence estimates adjusted for assay performance; the error level changes with the background seroprevalence. ConclusionsThe Sn. and Sp. of the serological assay are not fixed properties, but varying features depending on the testing population. To achieve precise population estimates and to ensure the comparability of seroprevalence, serosurveys should select assays with high performance validated not only by their manufacturers and adjust seroprevalence estimates based on assured performance data. More investigation should be directed to consolidating the performance of self-developed assays.
RESUMEN
COVID-19 is unlikely to be the last pandemic that we face. According to an analysis of a global dataset of historical pandemics from 1600 to the present, the risk of a COVID-like pandemic has been estimated as 2.63% annually or a 38% lifetime probability. This rate may double over the coming decades. While we may be unable to prevent future pandemics, we can reduce their impact by investing in preparedness. In this study, we propose RapiD_AI : a framework to guide the use of pretrained neural network models as a pandemic preparedness tool to enable healthcare system resilience and effective use of ML during future pandemics. The RapiD_AI framework allows us to build high-performing ML models using data collected in the first weeks of the pandemic and provides an approach to adapt the models to the local populations and healthcare needs. The motivation is to enable healthcare systems to overcome data limitations that prevent the development of effective ML in the context of novel diseases. We digitally recreated the first 20 weeks of the COVID-19 pandemic and experimentally demonstrated the RapiD_AI framework using domain adaptation and inductive transfer. We (i) pretrain two neural network models (Deep Neural Network and TabNet) on a large Electronic Health Records dataset representative of a general in-patient population in Oxford, UK, (ii) fine-tune using data from the first weeks of the pandemic, and (iii) simulate local deployment by testing the performance of the models on a held-out test dataset of COVID-19 patients. Our approach has demonstrated an average relative/absolute gain of 4.92/4.21% AUC compared to an XGBoost benchmark model trained on COVID-19 data only. Moreover, we show our ability to identify the most useful historical pretraining samples through clustering and to expand the task of deployed models through inductive transfer to meet the emerging needs of a healthcare system without access to large historical pretraining datasets.
RESUMEN
As machine learning-based models continue to be developed for healthcare applications, greater effort is needed in ensuring that these technologies do not reflect or exacerbate any unwanted or discriminatory biases that may be present in the data. In this study, we introduce a reinforcement learning framework capable of mitigating biases that may have been acquired during data collection. In particular, we evaluated our model for the task of rapidly predicting COVID-19 for patients presenting to hospital emergency departments, and aimed to mitigate any site-specific (hospital) and ethnicity-based biases present in the data. Using a specialized reward function and training procedure, we show that our method achieves clinically-effective screening performances, while significantly improving outcome fairness compared to current benchmarks and state-of-the-art machine learning methods. We performed external validation across three independent hospitals, and additionally tested our method on a patient ICU discharge status task, demonstrating model generalizability.
RESUMEN
Seroprevalence studies have been used throughout the COVID-19 pandemic to monitor infection and immunity. These studies are often reported in peer-reviewed journals, but the academic writing and publishing process can delay reporting and thereby public health action. Seroprevalence estimates have been reported faster in preprints and media, but with concerns about data quality. We aimed to (i) describe the timeliness of SARS-CoV-2 serosurveillance reporting by publication venue and study characteristics and (ii) identify relationships between timeliness, data validity, and representativeness to guide recommendations for serosurveillance efforts. We included seroprevalence studies published between January 1, 2020 and December 31, 2021 from the ongoing SeroTracker living systematic review. For each study, we calculated timeliness as the time elapsed between the end of sampling and the first public report. We evaluated data validity based on serological test performance and correction for sampling error, and representativeness based on use of a representative sample frame and adequate sample coverages. We examined how timeliness varied with study characteristics, representativeness, and data validity using univariate and multivariate Cox regression. We analyzed 1,844 studies. Median time to publication was 154 days (IQR 64-255), varying by publication venue (journal articles: 212 days, preprints: 101 days, institutional reports: 18 days, and media: 12 days). Multivariate analysis confirmed the relationship between timeliness and publication venue and showed that general population studies were published faster than special population or health care worker studies; there was no relationship between timeliness and study geographic scope, geographic region, representativeness, or serological test performance. Seroprevalence studies in peer-reviewed articles and preprints are published slowly, highlighting the limitations of using the academic literature to report seroprevalence during a health crisis. More timely reporting of seroprevalence estimates can improve their usefulness for surveillance, enabling more effective responses during health emergencies.
RESUMEN
As patient health information is highly regulated due to privacy concerns, the majority of machine learning (ML)-based healthcare studies are unable to test on external patient cohorts, resulting in a gap between locally reported model performance and cross-site generalizability. Different approaches have been introduced for developing models across multiple clinical sites, however no studies have compared methods for translating ready-made models for adoption in new settings. We introduce three methods to do this - (1) applying a ready-made model "as-is"; (2) readjusting the decision threshold on the output of a ready-made model using site-specific data; and (3) finetuning a ready-made model using site-specific data via transfer learning. Using a case study of COVID-19 diagnosis across four NHS Hospital Trusts, we show that all methods achieve clinically-effective performances (NPV >0.959), with transfer learning achieving the best results (mean AUROCs between 0.870-0.925). Our models demonstrate that site-specific customization improves predictive performance when compared to other ready-made approaches.
RESUMEN
BackgroundThe SARS-CoV-2 Delta variant has been replaced by the highly transmissible Omicron BA.1 variant, and subsequently by Omicron BA.2. It is important to understand how these changes in dominant variants affect reported symptoms, while also accounting for symptoms arising from other co-circulating respiratory viruses. MethodsIn a nationally representative UK community study, the COVID-19 Infection Survey, we investigated symptoms in PCR-positive infection episodes vs. PCR-negative study visits over calendar time, by age and vaccination status, comparing periods when the Delta, Omicron BA.1 and BA.2 variants were dominant. ResultsBetween October-2020 and April-2022, 120,995 SARS-CoV-2 PCR-positive episodes occurred in 115,886 participants, with 70,683 (58%) reporting symptoms. The comparator comprised 4,766,366 PCR-negative study visits (483,894 participants); 203,422 (4%) reporting symptoms. Symptom reporting in PCR-positives varied over time, with a marked reduction in loss of taste/smell as Omicron BA.1 dominated, maintained with BA.2 (44%/45% 17 October 2021, 16%/13% 2 January 2022, 15%/12% 27 March 2022). Cough, fever, shortness of breath, myalgia, fatigue/weakness and headache also decreased after Omicron BA.1 dominated, but sore throat increased, the latter to a greater degree than concurrent increases in PCR-negatives. Fatigue/weakness increased again after BA.2 dominated, although to a similar degree to concurrent increases in PCR-negatives. Symptoms were consistently more common in adults aged 18-65 years than in children or older adults. ConclusionsIncreases in sore throat (also common in the general community), and a marked reduction in loss of taste/smell, make Omicron harder to detect with symptom-based testing algorithms, with implications for institutional and national testing policies. SummaryIn a UK community study, loss of taste/smell was markedly less commonly reported with Omicron BA.1/BA.2 than Delta SARS-CoV-2 infections, with smaller declines in reported shortness of breath, myalgia and fatigue/weakness, but increases in sore throat, challenging symptom-based testing algorithms.
RESUMEN
Machine learning is becoming increasingly prominent in healthcare. Although its benefits are clear, growing attention is being given to how machine learning may exacerbate existing biases and disparities. In this study, we introduce an adversarial training framework that is capable of mitigating biases that may have been acquired through data collection or magnified during model development. For example, if one class is over-presented or errors/inconsistencies in practice are reflected in the training data, then a model can be biased by these. To evaluate our adversarial training framework, we used the statistical definition of equalized odds. We evaluated our model for the task of rapidly predicting COVID-19 for patients presenting to hospital emergency departments, and aimed to mitigate regional (hospital) and ethnic biases present. We trained our framework on a large, real-world COVID-19 dataset and demonstrated that adversarial training demonstrably improves outcome fairness (with respect to equalized odds), while still achieving clinically-effective screening performances (NPV>0.98). We compared our method to the benchmark set by related previous work, and performed prospective and external validation on four independent hospital cohorts. Our method can be generalized to any outcomes, models, and definitions of fairness.
RESUMEN
BackgroundOur understanding of the global scale of SARS-CoV-2 infection remains incomplete: routine surveillance data underestimates infection and cannot infer on population immunity, there is a predominance of asymptomatic infections, and uneven access to diagnostics. We meta-analyzed SARS-CoV-2 seroprevalence studies, standardized to those described in WHOs Unity protocol for general population seroepidemiological studies, two years into the pandemic, to estimate the extent of population infection and remaining susceptibility. Methods and FindingsWe conducted a systematic review and meta-analysis, searching MEDLINE, Embase, Web of Science, preprints, and grey literature for SARS-CoV-2 seroprevalence published between 2020-01-01 and 2022-05-20. The review protocol is registered with PROSPERO, (CRD42020183634). We included general population cross-sectional and cohort studies meeting an assay quality threshold (90% sensitivity, 97% specificity; exceptions for humanitarian settings). We excluded studies with an unclear or closed population sample frame. Eligible studies - those aligned with the WHO Unity protocol - were extracted and critically appraised in duplicate, with Risk of Bias evaluated using a modified Joanna Briggs Institute checklist. We meta-analyzed seroprevalence by country and month, pooling to estimate regional and global seroprevalence over time; compared seroprevalence from infection to confirmed cases to estimate under-ascertainment; meta-analyzed differences in seroprevalence between demographic subgroups such as age and sex; and identified national factors associated with seroprevalence using meta-regression. The main limitations of our methodology include that some estimates were driven by certain countries or populations being over-represented. We identified 513 full texts reporting 965 distinct seroprevalence studies (41% LMIC) sampling 5,346,069 participants between January 2020 and April 2022, including 459 low/moderate risk of bias studies with national/sub-national scope in further analysis. By September 2021, global SARS-CoV-2 seroprevalence from infection or vaccination was 59.2%, 95% CI [56.1-62.2%]. Overall seroprevalence rose steeply in 2021 due to infection in some regions (e.g., 26.6% [24.6-28.8] to 86.7% [84.6-88.5%] in Africa in December 2021) and vaccination and infection in others (e.g., 9.6% [8.3-11.0%] to 95.9% [92.6-97.8%] in Europe high-income countries in December 2021). After the emergence of Omicron, infection-induced seroprevalence rose to 47.9% [41.0-54.9%] in EUR HIC and 33.7% [31.6-36.0%] in AMR HIC in March 2022. In 2021 Quarter Three (July to September), median seroprevalence to cumulative incidence ratios ranged from around 2:1 in the Americas and Europe HICs to over 100:1 in Africa (LMICs). Children 0-9 years and adults 60+ were at lower risk of seropositivity than adults 20-29 (p<0.0001 and p=0.005, respectively). In a multivariable model using pre-vaccination data, stringent public health and social measures were associated with lower seroprevalence (p=0.02). ConclusionsIn this study, we observed that global seroprevalence has risen considerably over time and with regional variation, however around 40 % of the global population remains susceptible to SARS-CoV-2 infection. Our estimates of infections based on seroprevalence far exceed reported COVID-19 cases. Quality and standardized seroprevalence studies are essential to inform COVID-19 response, particularly in resource-limited regions.
RESUMEN
BackgroundRisk of bias (RoB) assessments are a core element of evidence synthesis but can be time consuming and subjective. We aimed to develop a decision rule-based algorithm for RoB assessment of seroprevalence studies. MethodsWe developed the SeroTracker-RoB algorithm. The algorithm derives seven objective and two subjective critical appraisal items from the Joanna Briggs Institute Critical Appraisal Checklist for Prevalence studies and implements decision rules that determine study risk of bias based on the items. Decision rules were validated using the SeroTracker seroprevalence study database, which included non-algorithmic RoB judgements from two reviewers. We quantified efficiency as the mean difference in time for the algorithmic and non-algorithmic assessments of 80 randomly selected articles, coverage as the proportion of studies where the decision rules yielded an assessment, and reliability using intraclass correlations comparing algorithmic and non-algorithmic assessments for 2,070 articles. ResultsA set of decision rules with 61 branches was developed using responses to the nine critical appraisal items. The algorithmic approach was faster than non-algorithmic assessment (mean reduction 2.32 minutes [SD 1.09] per article), classified 100% (n=2,070) of studies, and had good reliability compared to non-algorithmic assessment (ICC 0.77, 95% CI 0.74-0.80). We built the SeroTracker-RoB Excel Tool which embeds this algorithm for use by other researchers. ConclusionsThe SeroTracker-RoB decision-rule based algorithm was faster than non-algorithmic assessment with complete coverage and good reliability. This algorithm enabled rapid, transparent, and reproducible RoB evaluations of seroprevalence studies and may support evidence synthesis efforts during future disease outbreaks. This decision rule-based approach could be applied to other types of prevalence studies.
RESUMEN
BackgroundUncertainty in patients COVID-19 status contributes to treatment delays, nosocomial transmission, and operational pressures in hospitals. However, typical turnaround times for batch-processed laboratory PCR tests remain 12-24h. Although rapid antigen lateral flow testing (LFD) has been widely adopted in UK emergency care settings, sensitivity is limited. We recently demonstrated that AI-driven triage (CURIAL-1.0) allows high-throughput COVID-19 screening using clinical data routinely available within 1h of arrival to hospital. Here we aimed to determine operational and safety improvements over standard-care, performing external/prospective evaluation across four NHS trusts with updated algorithms optimised for generalisability and speed, and deploying a novel lab-free screening pathway in a UK emergency department. MethodsWe rationalised predictors in CURIAL-1.0 to optimise separately for generalisability and speed, developing CURIAL-Lab with vital signs and routine laboratory blood predictors (FBC, U&E, LFT, CRP) and CURIAL-Rapide with vital signs and FBC alone. Models were calibrated during training to 90% sensitivity and validated externally for unscheduled admissions to Portsmouth University Hospitals, University Hospitals Birmingham and Bedfordshire Hospitals NHS trusts, and prospectively during the second-wave of the UK COVID-19 epidemic at Oxford University Hospitals (OUH). Predictions were generated using first-performed blood tests and vital signs and compared against confirmatory viral nucleic acid testing. Next, we retrospectively evaluated a novel clinical pathway triaging patients to COVID-19-suspected clinical areas where either model prediction or LFD results were positive, comparing sensitivity and NPV with LFD results alone. Lastly, we deployed CURIAL-Rapide alongside an approved point-of-care FBC analyser (OLO; SightDiagnostics, Israel) to provide lab-free COVID-19 screening in the John Radcliffe Hospitals Emergency Department (Oxford, UK), as trust-approved service improvement. Our primary improvement outcome was time-to-result availability; secondary outcomes were sensitivity, specificity, PPV, and NPV assessed against a PCR reference standard. We compared CURIAL-Rapides performance with clinician triage and LFD results within standard-care. Results72,223 patients met eligibility criteria across external and prospective validation sites. Model performance was consistent across trusts (CURIAL-Lab: AUROCs range 0.858-0.881; CURIAL-Rapide 0.836-0.854), with highest sensitivity achieved at Portsmouth University Hospitals (CURIAL-Lab:84.1% [95% Wilsons score CIs 82.5-85.7]; CURIAL-Rapide:83.5% [81.8 - 85.1]) at specificities of 71.3% (95% Wilsons score CIs: 70.9 - 71.8) and 63.6% (63.1 - 64.1). For 3,207 patients receiving LFD-triage within routine care for OUH admissions between December 23, 2021 and March 6, 2021, a combined clinical pathway increased sensitivity from 56.9% for LFDs alone (95% CI 51.7-62.0) to 88.2% with CURIAL-Rapide (84.4-91.1; AUROC 0.919) and 85.6% with CURIAL-Lab (81.6-88.9; AUROC 0.925). 520 patients were prospectively enrolled for point-of-care FBC analysis between February 18, 2021 and May 10, 2021, of whom 436 received confirmatory PCR testing within routine care and 10 (2.3%) tested positive. Median time from patient arrival to availability of CURIAL-Rapide result was 45:00 min (32-64), 16 minutes (26.3%) sooner than LFD results (61:00 min, 37-99; log-rank p<0.0001), and 6:52 h (90.2%) sooner than PCR results (7:37 h, 6:05-15:39; p<0.0001). Sensitivity and specificity of CURIAL-Rapide were 87.5% (52.9-97.8) and 85.4% (81.3-88.7), therefore achieving high NPV (99.7%, 98.2-99.9). CURIAL-Rapide correctly excluded COVID-19 for 58.5% of negative patients who were triaged by a clinician to COVID-19-suspected (amber) areas. ImpactCURIAL-Lab & CURIAL-Rapide are generalisable, high-throughput screening tests for COVID-19, rapidly excluding the illness with higher NPV than LFDs. CURIAL-Rapide can be used in combination with near-patient FBC analysis for rapid, lab-free screening, and may reduce the number of COVID-19-negative patients triaged to enhanced precautions ( amber) clinical areas.
RESUMEN
COVID-19 is a major, urgent, and ongoing threat to global health. Globally more than 24 million have been infected and the disease has claimed more than a million lives as of October 2020. Predicting which patients will need respiratory support is important to guiding individual patient treatment and also to ensuring sufficient resources are available. We evaluated the ability of six common Early Warning Scores (EWS) to identify respiratory deterioration defined as the need for advanced respiratory support (high-flow nasal oxygen, continuous positive airways pressure, non-invasive ventilation, intubation) within a prediction window of 24 hours. We show these scores perform sub-optimally at this specific task. Therefore, we develop an alternative Early Warning Score based on a Gradient Boosting Trees (GBT) algorithm that is able to predict deterioration within the next 24 hours with high AUROC 94% and an accuracy, sensitivity and specificity of 70%, 96%, 70%, respectively. Our GBT model outperformed the best EWS (LDTEWS:NEWS), increasing the AUROC by 14%. Our GBT model makes the prediction based on the current and baseline measures of routinely available vital signs and blood tests.
